Skip to content

feat(engine-mcp): a capability search finds a symbol by name and answers in one call (PRD-865) - #644

Merged
jonit-dev merged 1 commit into
developfrom
prd-865-capability-search
Nov 9, 2026
Merged

jonit-dev merged 1 commit into
developfrom
prd-865-capability-search

Conversation

@jonit-dev

@jonit-dev jonit-dev commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

PRD: docs/PRDs/done/PRD-865-capability-search-finds-by-name-and-answers-in-one-call.md (100%: 3/3 phases, 4/4 acceptance criteria)

One commit on origin/develop (e897bf827). Rebased after #633–#645 landed: the previous head carried nine pre-squash doc commits that are now on develop, which is what made the PR conflicting.

Also lands the develop red this PR was queued behind (AGENTS.md: fix a red on develop in the PR already in flight). develop's board has been red since #639/#640, which is why every merge group — including this PR's three attempts — ejected: golden-path-template (platformer) failed with TN_PLAYTEST_BRIDGE_MISSING because the platformer kit still booted /src/main.ts on the page thread while the golden path serves index.html and asserts threads on. The kit now has the worker entry pair the starter shipped in #574: index.html → src/worker-page.ts, src/worker.ts, main.html for the page-thread arm, and the two-entry build. Its game code is worker-safe (render/loading.ts guards its canvas behind typeof document). That is the only reason this PR is not strictly PRD-865's scope.

What changed

engine_search_capabilities now finds a capability by its exported symbol name and answers in one call.

  • A name is indexed. The tokenizer splits camelCase; a request that spells a name as a run of whole words leads with that symbol, case- and separator-insensitive (mergeparts, characterbody3d), including a node named inside a sentence (GPUParticles3D sparks, Area3D trigger). The lead scales with how much of the request the name accounts for and a longer name wins a tie, so GPUParticles3D beats the generic Particles while first person view with a weapon held in the hand still answers the camera, not the View symbol.
  • Names carry their own index (digits stripped: GPUParticles3D reads as gpu particles) and the summary is scored over it at low weight — that is what makes a thinly described entry (traa) findable. It stays apart from the prose index on purpose: folding it in cost SnowField its row in the recall budget.
  • A row answers. ICapabilitySearchResult gains signature, overrides and deprecated: a row is the detail minus situations/aliases. AGENTS.md, agent-docs/kit-router.md, the shared threenative-capabilities skill and all sixteen kit AGENTS.md/CLAUDE.md now say read every row and call engine_capability_detail only for a symbol the search did not return.
  • MCP text is compact JSON. Indentation cost 11.8% of every response; threenative engine_* … from a shell re-indents its own copy.
  • A frozen gold set and a two-arm A/B: 150 real queries from 26 .threenative/engine-mcp.log files, 100 dev / 50 heldout, 20 exact-symbol, held-out sha256 frozen in the fixture and in the PRD. Labels were written by a fresh opus process that read only the logs and the manifest; 58 rows it judged engine-unowned were dropped before selection.

A/B against ecd2a1a56 (arm A), frozen held-out set, measured on this merged tree

metric (held out, 50 rows) arm A arm B delta
recall@1 0.400 0.620 +0.220 (target +0.05)
recall@5 0.580 0.780 +0.200 (target +0.10)
MRR 0.475 0.689 +0.214
exact-symbol rows at rank 1 0/7 7/7 +7
queries leaving arm A's top 5 — 0 pass (≤ 1)
modelled tokens to answer (29 rows both answer) 2 884 1 537 -47% (target -30%)

Live arm, 10 briefs × 2 runs per arm (claude -p --model sonnet --strict-mcp-config, the arm's rule as the prompt prefix), three runs because the base arm varies with how many details the model opens:

live arm A arm B delta
MCP tool calls 51 / 45 / 53 20 -56% … -62%
MCP response bytes 182 555 / 167 620 / 177 322 116 496 / 116 496 / 114 660 -30% … -36%
correct-symbol rate 1.00 1.00 equal
payload only (reference stubs excluded) 132 055 / 121 432 / 130 244 98 688 / 98 688 / 96 480 -19% … -26%

The split is recorded in the PRD's Decisions: the row change is the smaller half (details no longer opened ≈ 1.65 KB per search, five complete rows ≈ 4.5 KB, because a row is ~900 B at p50, not the 416 B the PRD assumed), and compact serialization adds 11.8% on every response including the reference stubs. Both figures print on every harness run.

Review

Two independent opus reviews of this branch; every finding fixed.

  • Surfaces the first review caught that the change had left behind: the recorded engine-mcp-tools.json description, the shared threenative-capabilities skill, the tracked packages/core/mcp/engine-server.mjs bundle, and the sixteen scaffold trees (re-measured on the merged tree).
  • The name channel first matched a token subset (physics answered afterPhysics); the next shape was too strict (whole-query equality missed a node named in a sentence, and a rarity floor hid use rapier). The final rule has no rarity constant: spelling plus a share-scaled lead plus a specificity tie-break, all 20 gold exact-symbol rows and both reviewers' probe sets green.
  • The harness gates live bytes on the PRD's metric, prints the stricter payload-only figure beside it, and fails closed on a live run that did not complete.
  • CI then found two consequences the reviews could not: the kit router text pushed starter two words over its 1,200-word instruction budget (the line is now rows answer; engine_capability_detail only for omitted symbols, zero delta, full detail in the threenative-capabilities skill), and seven Unreal-vocabulary aliases are now recalled with the aliases stripped because their capabilities' summaries name the same words — flagged baselineAlreadyRecalled with a comment rather than deleted.

Gates

  • pnpm tsx scripts/capability-search-ab.ts --base ecd2a1a56 --live --runs 2 — exit 0, every gate passes; artifact artifacts/capability-search-ab.json (local, gitignored).
  • pnpm exec vitest run packages/engine-mcp — 153 passed, 1 skipped: the gold spec (20/20 names at rank 1, locked held-out floor 0.60 / 0.76), the 86 search cases, and both reviewers' probes.
  • pnpm exec vitest run scripts/__tests__/kit-conventions.spec.ts scripts/__tests__/mcp-tool-schemas.spec.ts scripts/__tests__/verify-registry-install.spec.ts packages/core/__tests__/mcp-install.spec.ts packages/create-threenative/__tests__/scaffold-mcp.spec.ts scripts/__tests__/sync-agent-docs.spec.ts — 166 passed; scaffold.spec.ts green with the merged-tree hashes.
  • pnpm caps:recall — 0.938053, unchanged (an earlier shape of this change cost snow.refill).
  • pnpm typecheck, pnpm lint (0 errors), pnpm check:docs, pnpm sync:agents --check, pnpm quality: green.
  • Pre-push ci:fast: lint and drift pass.
  • pnpm build on this worktree fails only in examples/spell-lab: TN_WASM_ENGINE_MISSING (a fresh worktree has no Wasm build; packages all build).

@jonit-dev jonit-dev added prd:75% PRD phases in, acceptance open prd:100% PRD phases and acceptance complete and removed prd:75% PRD phases in, acceptance open labels Nov 9, 2026
@jonit-dev
jonit-dev force-pushed the prd-865-capability-search branch from 44b8b89 to 3305634 Compare November 9, 2026 21:52
@jonit-dev
jonit-dev marked this pull request as ready for review November 9, 2026 21:53
@jonit-dev
jonit-dev enabled auto-merge November 9, 2026 21:53
@jonit-dev
jonit-dev force-pushed the prd-865-capability-search branch from 3305634 to 8c7787d Compare November 9, 2026 22:23
@jonit-dev
jonit-dev added this pull request to the merge queue Nov 9, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to a conflict with the base branch Nov 9, 2026
@jonit-dev
jonit-dev added this pull request to the merge queue Nov 9, 2026
@github-actions

github-actions Bot commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

The merge queue stopped on native-platforms / Scaffolded starter desktop artifact (linux-x64) (failure).

Fix, push, re-enqueue.

@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 9, 2026
@jonit-dev
jonit-dev added this pull request to the merge queue Nov 9, 2026
@github-actions

github-actions Bot commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

The merge queue stopped on golden-path-template (platformer) (failure).

Fix, push, re-enqueue.

@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 9, 2026
@jonit-dev
jonit-dev added this pull request to the merge queue Nov 9, 2026
@github-actions

github-actions Bot commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

The merge queue stopped on golden-path-template (platformer) (failure).

Fix, push, re-enqueue.

@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 9, 2026
…ers in one call (PRD-865)

The index read only `situations` and `aliases`, so an exact symbol name answered
`verdict: none`: `godrays`, `traa` and `mergeParts` are exported by the manifest
and were unfindable, while `engine_capability_detail` cost a call per hit for a
row that lacked only `signature`, `overrides` and `deprecated`.

- The tokenizer splits camelCase, and a request that spells a name as a run of
  whole words leads with that symbol: whole-name equality is case- and
  separator-insensitive (`mergeparts`, `characterbody3d`), a node name inside a
  sentence counts (`GPUParticles3D sparks`, `Area3D trigger`), and the lead
  scales with how much of the request the name accounts for, with a longer name
  winning a tie. A name mentioned in passing cannot outrank the mechanic the
  request is about.
- Names carry their own index (digits stripped, `GPUParticles3D` reads as
  `gpu particles`) and the summary is scored over it at low weight, which is
  what makes a thinly described entry like `traa` findable. Keeping it apart
  from the prose index is deliberate: folding it in cost `SnowField` its row in
  the recall budget.
- A row equals the detail minus `situations`/`aliases`. `AGENTS.md`,
  `agent-docs/kit-router.md`, the shared `threenative-capabilities` skill and
  all sixteen kit `AGENTS.md`/`CLAUDE.md` now say read every row and call the
  detail only for a symbol the search did not return.
- MCP text is compact JSON; indentation cost 11.8% of every response and the
  shell command re-indents its own copy.
- A frozen gold set and a two-arm A/B: 150 real queries from 26
  `.threenative/engine-mcp.log` files, 100 dev / 50 held out, held-out sha256 in
  the fixture and the PRD, 20 exact-symbol. Labels came from a fresh opus
  process that read only the logs and the manifest. `search-gold.spec.ts` locks
  the held-out floor; `scripts/capability-search-ab.ts` extracts the base source
  with `git archive`, measures both arms and runs a live `claude -p` arm.

Against ecd2a1a on the frozen held-out half: recall@5 0.580 -> 0.780,
recall@1 0.400 -> 0.620, 0 rows lost from the top 5, modelled tokens -47%.
Live: MCP calls -56..62%, MCP response bytes -30..36%, correct-symbol rate
unchanged, 0 failed runs.

Two independent opus reviews; every finding fixed, including the sixteen
scaffold trees re-measured on the merged tree.

Also lands the develop red this PR was queued behind: the platformer kit gets
the worker entry pair the starter shipped in #574 (`index.html` ->
`src/worker-page.ts`, `main.html` for the page-thread arm, `src/worker.ts`, and
the two-entry build). Its golden-path leg boots `index.html` and asserts threads
on, so without the worker page the scaffolded game never booted and the leg
reported `TN_PLAYTEST_BRIDGE_MISSING`. The platformer's game code is worker-safe
(`render/loading.ts` guards its canvas behind `typeof document`).
@jonit-dev
jonit-dev force-pushed the prd-865-capability-search branch from 8c7787d to 4a78221 Compare November 9, 2026 23:27
@jonit-dev
jonit-dev merged commit a6f3292 into develop Nov 9, 2026
33 of 36 checks passed
@jonit-dev

Copy link
Copy Markdown
Contributor Author

Merged as a6f32923b; the merged tree is byte-identical to the reviewed head 4a78221cc, and develop CI started clean.

One correction to the commit message, for the record: the platformer worker-entry port in this commit (index.html → src/worker-page.ts, src/worker.ts, main.html, the two-entry build) makes the kit match the starter's worker mode, which the golden-path leg needs — but it does not by itself cure the red this PR was queued behind. Post-fix run 42720653536 still reports the same failure on that leg:

golden-path-template (platformer)  RuntimeDiagnostics 0, runtimeReady: true, noConsoleErrors: true
code: TN_PLAYTEST_BRIDGE_MISSING

So the page boots and the engine reports ready; what is missing is __THREENATIVE_PLAYTEST_BRIDGE__ on the page. The starter leg passes only because the scenario it drives (playtests/assets.playtest.json) needs no semantic capabilities, so this looks like worker-mode bridge mirroring (packages/core/src/worker.ts MIRRORED_GLOBALS) rather than anything kit-specific. That is a separate defect, red on develop since 4a8abe0a, and it needs a local golden-path run with the Wasm engine to pin down.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

prd:100% PRD phases and acceptance complete

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant